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Abstract 

Evidence  in  a  recent  study  contrasting  team  performance  in  traditional  unoptimized  architectures 
to  performance  in  non-traditional  optimized  architectures  showed  higher  performance  in  the 
traditional  unoptimized  architectures.  This  was  the  case  despite  the  fact  that  the  optimization 
process  matched  the  architecture  to  the  mission  and  reduced  intemodal  coordination.  The 
experimenters  surmised  that  unfamiliarity  with  the  non-traditional  optimized  architectures  was  an 
important  reason  for  the  lower  performance  in  the  optimized  architectures.  The  present 
experiment  was  conducted  to  improve  upon  the  design  used  in  the  previous  experiment  so  that  a 
valid  comparison  could  be  made  between  a  traditional  non-optimized  architecture  and  a  non- 
traditional  optimized  one.  As  expected,  once  teams  were  afforded  sufficient  training  with  the 
non-traditional  optimized  architecture,  they  performed  higher  with  it  than  with  a  traditional 
unoptimized  architecture.  The  results  provide  support  for  a  simple  model  linking  architecture 
type  through  team  processes  to  performance. 

1  Introduction 

Command  and  Control  (C2)  organizations  are  forced  to  adapt  when  confronted  with  unexpected 
events  and  tactical  surprises.  Entin,  Serfaty,  and  Kerrigan  (1998)  note  that  effective  organizations 
are  able  to  alter  their  strategies  or,  if  necessary,  alter  their  C2  architecture  to  cope  with 
unexpected  changes  and  perform  the  mission.  It  is  further  surmised  that  some  architectural 
configurations  may  be  more  conducive  to  adaptation  and  change  than  others.  We  hold  to  a  simple 
model:  the  C2  architecture  or  organizational  structure  impacts  team  processes  and  team 
processes  affect  team  performance.  If  the  C2  architecture  provides  a  flexible  and  conducive 
environment  such  that  team  processes  are  facilitated  then  the  team  processes  will  foster  high  team 
performance.  Conversely,  if  the  C2  architecture  does  not  support  team  processes,  for  example  by 
constricting  or  inhibiting  them,  then  team  processes  will  be  degraded  and  result  in  low  team 
performance.  One  research  issue  that  has  come  under  scrutiny  is  whether  C2  architectures  can  be 
designed,  for  example  through  the  use  of  optimization  principles,  to  facilitate  team  processes  and 
hence  facilitate  high  team  performance. 

In  a  recent  study  (Entin  et  al.,  1998)  six-person  teams  performed  a  mission  using  a  traditional  C2 
architecture  to  familiarize  them  with  the  architecture  and  mission.  Each  team  was  then  required 
to  perform  two  other  similar  missions,  one  using  the  initial  traditional  C2  architecture,  but  with 
30%  less  resources,  and  another  using  one  of  two  non-traditional,  but  optimized  C2  architectures. 
One  of  the  non-traditional  architectures  was  optimized  for  the  mission,  for  loss  of  resources,  and 
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to  reduce  intemode  (inter-team  member)  coordination.  Due  to  the  optimization  it  was  felt  that 
only  a  four-person  team  would  be  needed  to  perform  the  mission.  The  other  architecture  was 
similar  to  the  traditional  architecture,  but  partially  optimized  for  the  mission  and  loss  of  resources. 
Because  of  the  partial  optimization  it  was  felt  that  only  a  five-person  team  would  be  required  to 
perform  the  mission. 

Entin  et  al.  (1998)  expected  process  and  performance  outcome  scores  to  be  higher  in  one  of  the 
two  optimized  architectures.  They  expected  this  even  though  the  two  optimized  architectures 
would  be  non-traditional  and  hence  less  familiar  to  the  teams  than  the  traditional  non-optimized 
architecture,  and  even  though  the  size  of  the  team  would  be  reduced.  They  argued  that  the 
optimized  architectures’  milieus  would  be  so  conducive  to  efficient  performance  of  the  mission 
tasks  that  problems  such  as  initial  unfamiliarity  with  the  architecture  and  a  smaller  team  would  be 
overcome.  The  findings,  however,  did  not  confirm  these  expectations.  The  researchers  had 
apparently  overestimated  the  benefit  of  optimization  and  underestimated  the  effects  of  team  size 
and  familiarity  with  the  architecture.  The  performance  outcome  was  highest  in  the  traditional 
non-optimized  architecture  followed  closely  by  the  partially  optimized  architecture  and  lowest  in 
the  non-traditional  optimized  architecture.  Patterns  for  process  measures  such  as  teamwork  and 
workload  were  consistent  with  the  performance  measures. 

Entin  et  al.  (1998)  speculated  that  conditions  under  the  non-optimized  traditional  architecture 
were  better  known  and  understood  than  conditions  in  the  non-traditional  architectures,  and  that  is 
why  process  and  performance  outcome  measures  were  higher  under  the  non-optimized  traditional 
architecture.  However,  the  optimization  of  the  non-traditional  architectures  had  also  confounded 
team  size  with  architecture  type  precluding  a  clear  attribution  of  the  findings. 

In  accordance  with  Entin  et  al.  (1998),  we  surmised  that  unfamiliarity  with  the  non-traditional  C2 
architectures  precluded  teams  from  benefiting  from  the  optimization.  The  current  experiment  was 
conducted  to  examine  the  effects  of  familiarity  with  an  architecture  on  performance  under  that 
architecture  and  to  address  the  team-size  confounding  problem.  We  hypothesized  that  if  teams 
were  given  sufficient  practice  with  each  architecture  to  make  them  equally  familiar,  the  non- 
traditional  optimized  architectures  would  prove  superior  to  the  traditional  non-optimized 
architecture.  We  further  hypothesized  that  once  the  architectures  were  on  equal  footing  regarding 
familiarization,  an  optimized  architecture  would  allow  a  smaller  team  to  accomplish  the  same 
mission  as  a  larger  team  performing  the  mission  with  a  non-optimized  architecture. 

2  METHOD 

2.1  Participants 

Sixty  military  officers  attending  the  Naval  Postgraduate  School,  Monterey,  CA,  were  randomly 
assigned  to  ten  teams. 

2.2  Scenario  and  Mission 

The  C2  scenarios  and  missions  performed  by  the  ten  teams  were  adapted  from  the  North  African 
“insertion  from  the  sea”  scenario  used  in  Entin  et  al.  (1998).  The  primary  mission  tasks  include 
demining  and  taking  two  beaches,  taking  and  holding  a  hill  overlooking  one  of  the  beaches, 
identifying  and  destroying  the  correct  bridge  to  prevent  enemy  forces  from  reinforcing  enemy 


troops  near  the  beach  heads,  taking  and  holding  an  air  field,  and  taking  and  holding  a  sea  port.  In 
addition  to  the  main  mission  tasks  the  team  must  contend  with  a  number  of  additional  tasks  that 
arise  at  unexpected  times  during  the  scenario  trial  (e.g.,  suppressing  enemy  artillery,  destroying 
FROG  missile  launchers,  destroying  enemy  aircraft,  destroying  enemy  armor).  As  in  the  previous 
experiments,  the  scenarios  were  hosted  by  the  DDD-III  simulator  (Kleinman  et  al.,  1996).  The 
DDD-III  is  a  distributed  client-server  simulation  that  provides  a  flexible  framework  in  which  to 
study  team  decision  making  and  performance  in  complex  situations. 


2.3  Independent  Variables 


Three  C2  architecture  types  were  used:  AO-6,  a  six-person  traditional  non-optimized,  similar  to 
the  AO  architecture  used  in  Entin  et  al.  (1998);  A 1-6,  a  six-person  non-traditional  optimized, 
developed  for  this  experiment;  and  Al-4,  a  four-person  non-traditional  optimized,  similar  to  the 
Al  architecture  used  in  Entin  et  al.  (1998).  Each  team  was  randomly  assigned  to  perform 
missions  using  two  of  the  three  architectures. 

2.4  Dependent  Variables 

Performance  outcome  and  teamwork  skills  were  assessed  in  a  manner  quite  similar  to  that 
employed  by  Entin  et  al.  (1998).  Two  trained  observers  with  military  experience  were  present  at 
each  trial  to  independently  assess  both  performance  and  teamwork  skills  using  specially  designed 
evaluation  instruments  (Entin  et  al.,  1998;  Entin  et  al.,  1997;  Entin  et  al.,  1994).  The  performance 
instrument  was  comprised  of  ten  behaviorally- anchored  scales  evaluating  performance  quality  of 
the  mission  tasks.  The  teamwork  instrument  was  comprised  of  seven  behaviorally-anchored 
scales  that  assessed  six  dimensions  of  teamwork:  team  orientation,  communication,  monitoring, 
feedback,  back-up,  and  coordination  behaviors.  Reliability  of  both  instruments  is  high  (Entin  et 
al.,  1998;  Entin  et  al.,  1997).  The  ratings  from  the  two  observers  were  compared,  any  large 
deviations  were  adjudicated  and  then  the  ratings  from  the  two  observers  were  averaged  to  form 
the  performance  outcome  and  teamwork  skills  measures. 

The  DDD-III  simulator  calculates  and  reports  to  the  participants  at  the  end  of  each  trial  two 
performance  measures.  “Mission”  is  an  assessment  of  the  percentage  of  mission  tasks  completed 
where  each  task  is  weighted  by  the  percent  of  resource  accuracy  used  to  perform  the  task. 
“Strength”  is  an  attrition  based  measure.  Every  time  the  enemy  is  able  to  afflict  some  damage  to  a 
team’s  forces,  the  loss  is  deducted  from  the  team’s  overall  force  strength.  The  percentage  of 
overall  force  strength  remaining  at  the  end  of  the  trial  is  the  strength  score. 

As  in  previous  studies  (Entin  et  al.,  1998;  Entin  et  al.,  1997;  Entin  et  al.,  1994)  two  trained  and 
experienced  observers  coded  in  real  time  all  the  communications  occurring  among  team  members. 
One  observer  focused  on  the  communications  uttered  by  half  the  participants  while  the  other 
observer  coded  the  utterances  of  the  other  half.  In  this  experiment,  however,  the  recording 
medium  used  by  observers  was  changed  substantially.  The  pencil  and  paper  recording  matrices 
were  replaced  with  hand-held  3Com  Pilot  III  computers.  Equipped  with  software  specifically 
designed  for  the  experiment,  the  touch  sensitive  screen  of  each  Pilot  displayed  the  recording 
matrix.  The  matrix  displayed  on  the  screen  was  the  same  as  the  paper  matrix  used  Entin  et  al. 
(1998).  The  two  trained  observers,  who  were  connected  to  the  communication  net  via  earphones, 
coded  the  communications  among  the  team  members  by  touching  the  appropriate  cell  of  the 
matrix  with  a  stylus.  The  software  recorded  the  cell  number  and  the  time  (i.e.,  number  of  seconds 
from  the  start  of  the  simulation).  Thus,  a  time  stamped  coding  of  communication  within  the  team 
was  obtained. 


2.5  Design  and  Procedure 


The  experiment  was  conducted  in  two  sessions.  During  the  first  three-hour  session  a  team 
engaged  in  two  45 -minute  practice  trials  using  the  architecture  scheduled  for  the  data  collection 
trial  and  then  completed  the  45-minute  data  collection  trial.  The  inclusion  of  two  pre-experiment 
practice  trials  permitted  teams  to  become  familiar  with  the  architecture  they  would  be  using  in 
that  experiment  trial.  This  process  was  repeated  again  for  the  other  architecture  the  team  was 
assigned  in  a  second  three-hour  session  that  took  place  one  to  five  days  later.  Following  the  last 
data  collection  trial,  each  team  engaged  in  an  after- action  review.  The  data  collection  trials  and 
the  after-action  reviews  were  videotaped. 

3  Results  and  Discussion 

3.1  Performance  Results 

Both  simulator  measures  showed  the  same  pattern  of  results,  however  the  strength  attrition  score 
proved  more  sensitive  and  is  the  simulator  based  assessment  used  in  this  experiment.  From  the 
observer-based  performance  outcome  measure  depicted  in  Fig.  1  and  the  simulator-based 
performance  measure  (strength  attrition  score)  shown  in  Fig.  2  we  can  see  that  teams  performed 
significantly  higher  (ps  <  .05)  under  the  A 1-6,  non-traditional  optimized  architecture  than  under 
the  other  architectures.  We  can  further  see  that  when  familiarity  with  the  various  architectures  is 
equalized,  optimization  prevails  over  the  traditional  architecture  as  well  as  team  size,  i.e.,  teams 
performing  with  the  AO- 6,  the  six-person  non-op timized  architecture,  performed  below  the  two 
optimized  architectures. 

Multivariate  and  univariate  analyses  of  the  items  comprising  the  performance  measure  show  that 
performance  on  three  tasks-  destroy  the  bridge,  destroy  the  FROG  launchers,  and  seize  the 
airfield/seaport-were  most  responsible  for  the  difference  in  performance  outcome  between  the 
AO-6  traditional  non-optimized  architecture  and  the  Al-6  non-traditional  optimized  architecture. 
When  the  ten  task  items  were  regressed  on  average  overall  performance  outcome,  performance 
on  the  tasks  destroying  the  FROG  launchers,  destroying  the  tanks,  and  taking  the  airfield/seaport 
tasks  accounted  for  93  %  of  the  variance  in  predicting  average  overall  performance  outcome. 
Two  tasks  emerge  in  both  these  analyses:  destroying  the  FROG  launchers  and  taking  the 
airfield/seaport.  Interestingly  a  team’s  performance  on  a  task  that  occurs  unexpectedly,  destroying 
the  FROG  launchers,  discriminates  between  the  AO-6  traditional  non-optimized  and  the  Al-6  non- 
traditional  optimized  architectures,  as  well  as  predicts  average  overall  performance  outcome.  The 
other  common  task  to  both  analyses  is  taking  the  airfield/seaport,  a  main  mission  task  and  the 
primary  objective  of  the  mission. 
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Figure  1 .  Observer  Based  Performance  Scores  for  the 
Three  Architectures 


Figure  2.  Simulator  Based  Performance  Scores  for  the  Three  Architectures 


3.2  Team  Process  Measures 

Results  from  the  observer-based  teamwork  measures  shown  in  Fig.  3  corroborate  the 
performance  findings.  Teamwork  scores  were  highest  when  teams  used  the  Al-6,  the  non- 
traditional  optimized  architecture,  and  lowest  when  the  AO-6,  non-optimized  architecture  was 
used  (ps  <  .02).  Four-node  teams  performing  with  the  Al-4  architecture  also  exhibited  higher 
teamwork  scores  than  the  six-node  teams  performing  with  the  AO-6  architectures  (ps  <  .05). 
Analysis  of  the  items  making  up  the  teamwork  instrument  showed  that  two  teamwork  skills, 
monitoring  and  coordination  behavior,  were  particularly  salient  in  discriminating  between  the 
optimized  and  non-optimized  architectures  as  well  as  in  predicting  the  overall  teamwork  score.  In 
the  latter  analysis,  monitoring  and  coordination  behavior  accounted  for  93%  of  the  variance  with 
the  overall  teamwork  score. 

Another  important  source  of  team  process  measures  is  team  communications.  Several  measures 
derived  from  the  communication  either  support  the  performance  findings  or  verify  a  link  between 
architecture  and  the  process  measures.  The  information  anticipation  ratio  assesses  how  well  team 
members  anticipate  other  team  members’  information  needs  and  push  information  to  them  before 
being  asked  for  it.  The  pattern  of  results  depicted  in  Fig.  4  indicates  that  the  information 
anticipation  was  highest  with  the  Al-6  architecture  and  lowest  with  the  AO-6  architecture  (ps  < 
.05).  For  team  members  to  successfully  anticipate  the  information  needs  of  others,  they  must 
have  an  accurate  mental  model  of  the  situation  and  other  team  members.  So  in  essence  the 
information  anticipation  results  indicate  that  the  non-traditional  optimized  architectures  foster  the 
development  of  better  mental  models  of  the  situation  and  other  team  members. 


Figure  3.  Observer  Based  Teamwork  Scores  for  the  Three  Architectures 
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Figure  4.  Information  Anticipation  Ratio  for  the  Three  Architectures 

Analysis  of  the  total  communications  rate  for  teams  reveals  that  the  communication  rate  under  the 
AO-6  architecture  was  significantly  higher  (ps  <  .05,  one-tail)  than  in  the  optimized  architectures. 
Further  analyses  also  showed  that  the  coordination  rate  and  the  request  for  resource  utilization 
were  significantly  higher  with  the  AO-6  architecture  than  with  the  two  optimized  architectures  (ps 
<  .02).  The  significance  of  these  findings  can  be  appreciated  when  one  considers  that  a  goal  of 
the  optimization  of  the  A 1-6  and  A 1-4  architectures  was  to  reduce  intemodal  communication  and 
coordination.  It  appears  that  the  optimization  process  achieved  its  goal. 

4  Conclusions 

This  experiment  was  conducted  to  improve  upon  the  design  used  in  the  experiment  described  in 
Entin  et  al.(1998)  so  that  a  valid  comparison  could  be  made  between  a  traditional  non-optimized 
architecture  and  non-traditional  ones  optimized  to  accommodate  the  mission.  Prior  experimental 
evidence  indicated  that  teams  tend  to  perform  highest  using  organizational  structures  they  are 
familiar  with  as  opposed  to  organizational  structures  that  are  unfamiliar,  even  if  the  unfamiliar 
structures  are  better  suited  to  the  mission.  To  reap  the  benefits  of  the  better  architecture,  teams 
must  train  with  and  come  to  understand  the  new  architecture.  Findings  from  the  prior  and  current 
experiments  clearly  support  the  necessity  of  practicing  with  an  unfamiliar  organizational  structure. 
In  Entin  et  al.(1998)  the  traditional  AO-6  architecture  fostered  superior  performance  over  the 
non-traditional  architecture  optimized  for  the  mission  tasks.  When  teams  were  afforded  time  to 
train  with  the  unfamiliar  non-traditional  optimized  architectures  the  superiority  of  these 


architectures  to  perform  the  mission  surfaced  and  teams  using  them  perform  significantly  higher 
than  teams  using  the  traditional  (non-optimized)  architecture. 

To  provide  an  explanatory  mechanism  for  performance  outcome  we  postulated  that  team 
processes  underlay  the  team’s  performance,  and  that  organizational  structure  affects  team 
processes.  Organizational  structures  that  support  important  team  processes  give  rise  to  high 
levels  of  team  performance.  When  organizational  structure  is  not  supportive  of  effective  team 
processes  or  is  counter-productive  to  effective  team  processes,  team  performance  suffers.  For 
each  of  the  team  processes  assessed-teamwork  skills,  workload,  organizational  awareness,  and 
communications-the  pattern  of  results  always  mirrored  the  performance  outcome  results.  We 
submit  that  superior  team  processes  were  the  foundation  for  high  team  performance.  Moreover, 
evidence  from  several  of  the  communication  process  measures  indicate  that  the  optimization  goals 
of  the  modeling  effort  were  achieved.  The  modeling  effort  strove  to  reduce  intemodal 
coordination  within  the  optimized  architectures.  The  findings  indicate  that  communications, 
coordination,  and  resource  request  utilization  were  significantly  lower  in  the  optimized  than  non- 
optimized  architectures.  Not  only  were  the  modelers  successful  in  constructing  the  desired 
architectures,  but  architecture  type  differentially  affected  team  processes,  as  hypothesized. 
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